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ABS TRACT 

As part of a symposium that examined the utility of 
adapting the techniques of cost-effectiveness analysis for curriculum 
and program planning in higher education, this paper examines the 
methodology and procedures of the technique. A checklist outlines 
some of the prdblems and potential and some guidelines are suggested 
for cost^ef fectiveness analysis. (Author/MLF) 



* Docum 

* materials 

* to obtain 

* reproduci 

* of the mi 

* via the E 

* responsib 

* supplied 
♦♦♦♦♦♦♦ *♦** 



***♦♦****♦♦♦♦♦♦♦ 

ents acquired by 

not available f 

the best copy a 

bility are often 

crofiche and har 

RIC Document Rep 

le for the quali 

by EDRS are the 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



ERIC include many informal unpublished * 

rom other sources. ERIC makes every effort * 

vailable. Nevertheless, items of marginal * 

encountered and this affects the quality * 

dcopy reproductions ERIC makes available * 

reduction Service (EERS) . EDRS is not * 

ty of the original document. Reproductions * 

best that can bei^made from the original. * 
♦♦♦♦^* ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦ 



ERLC 



•PERMISS:ON TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERiC) AND 
THE ERIC SYSTEM CONTRACTORS" 



US OEPAKTMENT OF HEALTH. 
EDUCATION A WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN REPRO* 
DUCEO EXACTLY AS RECEIVED FROM 
THE PERSON OR OAGaNi2aTiON ORIGIN* 
ATiNGiT POINTS OF VIEW OR OPINIONS 
STATED 00 NOT NECESSARILY REPRE* 
SENT OF FiClAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLiCY 



COST-EFFECTIVENESS ANALYSIS PITFALLS 



and POTENTIALS FOR PLANNING INSTRUCTIONAL PROGRAMS 



Philip Doughty 
Syracuse University 
April 1977 



A paper presented at the American Educational Research Association 
Annual meeting - April 1977, New York City 
Session #19.09 




Overview 

The instructional planner, depending upon prior education and 
experience and current context, may owe primary professional allegiance 
to any number of associations and academic disciplines* Descriptors 
suca as educational researcher, Tevaluator, economic analyst, instructional 
designer, operations researcher, curriculum specialist and systems analyst 
represent some of the monikers used by professionals who perform the 
instructional planning function* 

Regardless of race, creed or professional affiliation, planners 
making or informing decisions must consider issues related to costs 
and effectiveness. How and when these issues are .considered are be- 
coming major concerns in these times of retrenchment and resource 
reallocation. 

The intent of this paper is not to provide an all-purpose recipe 
for the conduct of cost-effectiveness analysis. Attempts at such a cook- 
book approach would^-likely result in something akin to. gastronomic distress. 
VJhat has been attempted in fields outside conventional education contexts 
is the articulation of cost-effectiveness guidelines that include issues, 
outlines, examples and references (Doughty, Stern and Thompson, 1976). 
To date, there is little evidence that Betty Crocker has responded to the 
'challenge. 

Ultimately the question is asked: 'U^here does the planner begin in 
^developing a cost-effectiveness analysis perspective?" This paper will 
outline several issues that distinguish studies that have incorporated this 
perspective from those that have not. This is definitely not the first 
article to suggest that you start by, identifying perils and pitfalls of 
cost-effectiveness thus providing a kind of ten commandments as a spring- 
board. (Kazanowski, 1968; Quade 1976) 
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One of the major problems is the perception that Cost-Effectiveness 
analysis does not require, any particular technical or professional competence, 
but that there is a requirement for copious quantities of common sense and 
Sound logic. Considerable and continuing support tor this notion is supplied 
through the proclamations of decision-makers in most every field. Written 
and verbal utterances 'such as: "We developed it in a cost-effectiveness 
manner" or "The benefits were maximized while the costs v:ere cut 20 percent" 
or "Our benefit-cost ratios prove..." or "The decision was made on the 
basis of cost-effective reasoning" all serve to perpetuate the illusion of 
science and truth. 

Somewhat farther along this art-science continuum falls a relatively 
select but growing number of professionals who mask, their more sophisticated 
blunders with verbage and jargon peculiar to their particular field of inquiry. 
Most of these are the results of errors in logic or f alseassumptions. Economic 
analysts, summative evaluators, cost analysts, program budgeters and operations 
researchers all have high potential for inclusion in. this club. An even more 
unfortunate situation arises when a quality piece of work is obfiscated by 
the author in catering to external requirements for specialized language, 
format or purpose. Such is often the case with parochial professional 
journals and research oriented rather than decision oriented dissertation 
committees. 

The following checklist outlines many but obviously not all of the 
issues that should be considered* Each of the issues is discussed in 
brief sections that follow. 
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COST-EFFECTIVENESS ANALYSIS SELF-CHECK 



Pitfalls for Evaluators, Analysts and Instructional Planners 



That's Not 
Evaluative Criteria -^2 Maybe Not Us 



i: 


Poorly Defined Decision Rules or Criteria 


1. 


2. 


Select Wrong Criteria 


' 2. 


3. 


Single Criterion 


3. 


4. 


Emphasize Thruput Indicators 


4. 


5. 


Ignore Spillover or Unanticipated Effects 


5. 


6. 


Overquant if ication 


6". 


7. 


Overgeneralizing Results 


7. 



B. Cost Issues 

1. Rely on Jurisdictional Cost Data - ^ 1. 

2. Undue Emphasis on Total Dollar Cost Criterion 2. 

3. Include Sunk Costs in Cost Analysis 3. 

4. Assume Unreasonable Depreciation' Schedules 4, 

5. Focus Only on Dollar Expenditures 5. 

C. Cost and Effectiveness Relationships 

1. Misuse of Ratios 1. 

2. Failure to Fix Cost or Effectiveness 2, 

3. Assume or Imply Causal Relationships 3, 
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Evaluative Criteria 

1. P.oorly Defined Decision Rules or Criteria ; Studies conducted to 
aid 'decision making oftentimes are well conducted arid reported but 
fail to have any discernable impact. More often then not,^the public 
and private criteria used by key decision makers have not been 
determined' nor used as primary components in the initial design. 

Of course, at times it is not in the interest of an individual or an 
institution to divulge some decision rules. In these instances, the 
^ analyst's role and potential impact are considerably weakened. 

2. Select Wrong Criteria : It is often tempting for analysts to select 
criteria that can be easily defined and/or quantified, but are irrelevant 
to the decision. On the otTier hand, criteria are also selected 
because of their* obvious potential impact, but they too may be the 
wrong or educationally insignificant ones. Dollar cost measures 

are as likely to be improperly or insufficiently reported as are out- 
come measures or indicators, 

3. Single Criterion t."*^ How often have reviewed studies that report 
results of no significant difference when in fact there were highly 
significant differences in other variables or criteria? No differences 
in dollar costs to an institution for ^instance may ignore a require- 
ment for the expenditure of extensive out-of-class time by learners. 

To rely upon a single criterion, however encompassing it may be, places 
considerable faith in the analyst to select or create the ultimate-all 
purpose measure or indicator that communicates to all. 

4. Emphasize Thruput Indicators : In the absence of utlimate criteria, 
based upon needs, job analysis, performance requirements or whatever, 
proxy measures are oftentimes used as indicators or substitutes. 

This is of course acceptable practice. However, .misleading conclusions 
ere often drawn when thruput indicators such as cost-per-student-credit- 
hour data are used as primary evidence of quality or efficiency rather 
than as idiosyncratic, unccmparable ratios of thruput. 
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A. Evaluative Criteria continued 

5. Ignore Spillover or Unanticipated Effect s: A myopic fixation on 
established criteria excluding consideration of other direct*' 
evidence may not identify the alternative that compares 
favorably on selected criteria but excells or fails miserably 

in some additional area. Comments such as: "Learners came 
in apprehensive and left hostile" or "The reading instruction 
system x^as very efficient. Isn't it a shame the students are' 
no longer interested in going to the library?" provide insights 
into effects that may impact on systems well outside the one being 
considered. Obviously consideration of attribution and causality 
issues is always required but to ignore "other effects" shows ev-dence 
of limited vision. 

6. Overquantif ication ; In the quest to demonstrate the science of Cost- 
Effectiveness analysis, many individuals and disciplines require that 
all criteria be quantified in some fashion. Never mind the concern 
that in so doing, the analyst has now become decision-maker by 
providing standards of value and v/orth and utility rather than 
reserving those perogatives for the consumer (s). 

7. Overgeneralizing Results ; As generally applied, Cost-Effectiveness 
studies are not designed to generate universal or broadly generalizable 
results. Studies are conducted to inform and perhaps influence 
organizational behavior, or at times, invididuals. Expectations of 
theory driven, empirically validated generalizable models of cost- 
effectiveness analysis are unrealistic given the purpose and focus 

of this process. 



B. Cost Issue 

!• Rely on Jurisdictional Cost Data ; Most primary sources of dollar cost 
information aggregate and report those data according to "authority 
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Cost Issue continued , 

c 

1. Arezis" or ''domains of responsibility" which oftentimes do not 
represent program, system or result areas. Jurisdictional costs 
provide accounting information .to ensure budgetary compliance 
but rarely do they directly relate to outcomes or jprogram goals. 
Generally spejiking, program or functionally related cost data are not 
available from existing records and must be collected or projected 
by the analyst. ^ 

2. Undue Emphasis on T6tal Dollar Cost Criterion : A false asstinirtion 
often made or implied in reports is that the total woUar coQt of 

■an alternative, including design, development, implenaentatiuu, anC 
lifetime operation, represents the total spectrum of the negative 
dollar costs. For instance, it may be the case that the salary of 
a master instructor, skillful programmer, or competent manager can 
be accurately reflected, it is another issue entirely to assume that 
\ simply listing such salaries ensures thfeir availability. In addition, 
reporting only total system costs does not permit sci:ijj|iny of cost- 
time phasing issues such as high front-end loading vs» more costly 
continuing operational expenditures. 

3. Include Sunk Co^ts in Cost Analysis : When comparing feasible 
alternatives, one option may well be to continue existing system 
that has already been Implemented. In this case, an equitable' 
analysis would only consider the future costs of operating and 
maintaining that system, not the sunk costs that were allocated in 
the past. In their attempt to be comprehensive, overzealous analysts 
often include such costs rather than limiting the data to those 

that relate to future expenditures and outcomes. 

4. Assume Unreasonable Depreciation Schedules ; One way to, distribute the 
cost of large initial hardware expenditures is to depreciate -ttiem over 
some specified number of years. Combining the notions of technological 
and coritent obsolescence with time or use-based depreciation is also 
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B; Cost Issue continued * 

4. legitimate and recommended. Problem'^ arise, however, when 
analysts make unwarranted assumptions about lifespans of comp.uter 
and viddo systfems or course content thus making it easier to 
justify high front-end expenditures. This is one area where ^careful 
judgment and public assumptions are basic requirements. A different . 
but related' issue concerns the assumptions made about the nisaoer of/ 
cycles an instructional program or system will be offered as well as 
the number of learners or participants served during' the projected 
lifetime of an alternative. These items have tremendous, almost • - 
overpowering impact on decisions related to judging economy of scale. 
Judicious estimates with supporting rationale help offset the. healthy 
skeptism decision makers have about such data. 

5. Focus Only on Dollar Expenditures ; Economists are quick to point out 
that costs are not simply negative consequences of a decision. There 
are, however, many decisions or alternatives that have cost implica- 
..tions of some kind for a variety of audiences. Negative benefits such 

as low student and faculty morale, study time diverted from other 
courses, and professional time diverted from other scholarly endeavors 
are examples of the broad range of costs that are not represented by 
balance sheets and voucher records. " / 



Cost and Effectiveness Relationships 

1. Misuse of Ratios ; Some analyst argue that the ultimate in scientific 
approaches to Cost--Ef f ectiveness analysis is to quantify the criteria, 
'preferably in dollar terms', so that benefits to cost ratio comparisons 
can be made. This approach combines the overquantif ication pitfall 
with a more fundamental problem - the lack of attention to the size 
of magnitude of the numerators and denominators. Most decision-makers 
are capable of selecting- betting odds of 3 to 1 over a ratio of 5 to 2. 
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Cost and Effectiveness Relationships continued • . . 

!• Of course what wasn't considered was the additional information 
that 3:1 was actually $30 to $10 an4 the 5:2 indicated a benefit 
with '5k million dollars fey: a capital investment of 2 million 
dollars. '^lu this case, 'not ^nly did the ratio's dictate the decision, 
the initial selection was limly the wrong one. ^ 

2. Failure to. Fix Costs or Effectiveness' : Literature from many fields 
abounds with the -notion that cost-effectiveness analysis is' a process 
that identifies the alternative which requires minimal resources 

and provides maximum effectiveness. It can be argued that this is 
Based upon the assumption that if you search hard enough or design 
creatively then you will eventually discover the alternative that* 
provides infinite effectiveness for zero cost. Briefly, a more tenable 
guideline reads as follows: a) identify a desired level of effectivenes 
and then examine the cost of alternative means of achieving that level, 
or b) specify a budget level and examine the level(s) of effectiveness 
that might be achieved through different alten'iatives. Seldom is 
either 'of these represented in the literature. As a minimiim gesture, 
analysts should underscore the considerable problems associated with 
comparing alternatives with unlike. resource requirements and levels^ 
of effectiveness. 

3. Assume or Imply Causal Relationships : To specify, measure, and report' 
^ resource requirements, process descriptions and system outcomes is 

a legitimate responsibility of the analyst. However, to Imply or 
assume that direct causal relationships exist between these variables 
requires a technical and conceptual leap well beyond current 
Olympic standards. Cost-Effectiveness reports should always contain 
caveats concerning correlation, causality, and attribution. 
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. . Next Steps - - 

• - ' r 7^ 

Instructional planners interested in pursuing Cost-Effectiveness 
issues and procedures are likely to be initially encouraged and then 

• 4 

•dismayed. Many Cost-rEEf ectiveness ^elated studies, articles, and texts 
Imve been published, in the past six to seven years* None of them proclaim 
to be the comprehensive guide jcor instructional or educational planners/ 
In addition, cdrvaiderable translation* of terminology and concepts is 
required. Expl'ai)atipns and" examples contained in* the following publicatio'tis 
are particularly relevant, require less translation than most, and merit 
consideration. - ^ • • ' • . 

1^ Doughty,- P. •& Bcilby, A. Cost Analysis and Jeacher EdUcatioa: _ 
' < ' ♦ A Comment on Relevanu Relationships, A Review of Existing 
Models. "Syracuse 'University, 1974'. (Monogr*aph) . 

2. Doughty, P., Stem, H. i Thompson, C. Guidelines for Cost- 

' Effectiveness Analysis for Navy Training and Education . 
Navy Personnel Research and Development Center, San Diego, 
California. July, 1976. 

3. English, J. Morley (ed.) Cost- E ffect iven^Bs: The Economic 

Evaluation of Engineered Sy^^cems . New York: John Wiley 
& Sons Inc., 1968. 

4. Kazanowski, A. D. "Cost-Effectiveness Fallacies and Miscon- 

ceptions Revisited" in English, J. Morley (ed.). ^ 

5. Levin, Henry M. "Cost Effectiveness' Analysis in Evaluation 

Research" in Guttentag M. and Struening E. 
^ Handbook of Evaluation Research . Beverly Hills: 
SAGE Publications, 1975. 

6. Quade, E. S. Analysis for Public Decisions . New York, 

American Elsevier, 1976. (especially Chapter 20 - 
"Pitfalls and Limitations"). 



